Weighted Policy Constraints for Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to learn policy from the passively collected offline dataset. Applying existing RL methods on static dataset straightforwardly will raise distribution shift, causing these unconstrained fail. To cope with shift problem, a common practice in is constrain explicitly or implicitly close behavioral policy. However, available usually contains sub-optimal inferior actions, constraining near all actions make inevitably behaviors, limiting performance of algorithm. Based this observation, we propose weighted constraints (wPC) method that only constrains learned desirable making room for improvement other parts. Our algorithm outperforms state-of-the-art algorithms D4RL gym datasets. Moreover, proposed simple implement few hyper-parameters, wPC robust low computational complexity.